Overview

I want to explain how the Lego sorting machine is going to work on a conceptual scale. We have a large number of different components in the system, so lets start with a diagram.

The olive colored components are different pieces of software running on a computer. The grey components are pieces of hardware, and the green one is a microcontroller:

Components:

• Hopper - a large bin which feeds bricks onto the belt
• Camera - a webcam with lights to illuminate parts as they go by
• Bins - a number of bins arranged along the length of the belt
• Air valves - pneumatic valves with are positioned to blow bricks into the bins as they pass by
• End of Belt - a bin to catch bricks that go off the end of the belt
• Taxidermist - a program on the server which captures pictures and trims/crops/scales them
• MT Mind - a neural network that is trained to identify lego parts from pictures
• Classifist - a custom program to decide which bin each part belongs in
• SUIP - (Sorting User Interface Platform) a graphical front end for operating the machine
• Belt Buckle - an Arduino which controls the air valves and drives the belt
• Conveyor Belt - a conveyor belt

But how does it work?

The machine works by feeding parts from the hopper onto the conveyor belt. This will be achieved by some mystical device that Jamie and I have yet build. We currently believe it will be the most difficult aspect of this entire operation due to the wide variety in shape and size of Lego parts. As the parts ride along the belt, they pass in front of the camera, and have their picture taken.

The Taxidermist

The Taxidermist watches a video feed from the camera for a new part to enter. When a part passes by, it will immediately send a time-stamp to the Belt Buckle, signaling that a part is on its way down the belt. Now the taxidermist will take an image of the part, crop and resize it, and send it to the MT Mind. We have a functional pre-alpha version of this program, that processes video at around 30fps. Jamie wrote it using python and OpenCV.

The MT Mind

The MT Mind is a convolutional neural network.  The MT Mind will review the picture submitted by the Taxidermist and determine exactly which part (by its official part number) is in front of the camera. The part number, along with the timestamp from the Taxidermist, is then passed to the Classifist. In its current state, the MT Mind can classify bricks and plates with about 95% accuracy. Not great, but its a solid first attempt.

The Classifist

The classifist is more than just a ridiculous play on words. It takes the part number provided by the MT Mind and checks it against a config file which will determine the bin that part belongs in. The config file is setup by the operator before the machine runs, and will allow for sorting by part number, part category, and more. Once the classifist has made a decision, it will send the timestamp and a bin number to the Belt Buckle.

The Belt Buckle

In the time that has passed since part was first seen by The Taxidermist, the part will have moved some distance down the belt. The Belt Buckle has a sensor that tracks the movement of the belt, and it keeps track of each part's position from the moment the camera sees it. By the time the Classifist makes a decision, the belt buckle already knows about the part and is tracking its progress down the conveyor. When the part reaches its intended bin, an air valve blows it in. The real trick here is to ensure that the sorting software does it's job before the part passes the first bin.

The Belt Buckle is an Arduino, specifically the Mega. I chose the mega because it has 4x as much memory as my older Duemilanove model and many more I/O pins.

Currently, the Belt Buckle is housed in a cute plastic box with a joystick on top:

 We haven't exactly decided what the joystick is for yet. But it's pretty sweet nonetheless.

The SUIP

Nextly, we have the SUIP(yes, it's pronounced like "soup"). The Sorting User Interface Platform will be a graphical interface which will connect to the classifist. It will allow for easy changes to the sorting configuration, and also display logs and error messages. It currently does not exist, but I'm sure Jaden Smith already believes in it.

The Conveyor Belt

The conveyor belt was acquired from eBay. It's already the best purchase I have ever made. Even if the sorting project ends in failure and dismemberment, this conveyor belt will be put to work somehow. I might mount it next to my computer to feed me an constant stream of nachos while I work, or attach steps to it and prop it up on a 45 degree angle to make a perpetual slinky machine. (I feel the need to shout out to Matthias Wandel over at woodgears.ca, this guy is awesome and you should watch all his videos.)

The Breaks

It should be pointed out that at his phase, some(most) of what you just read is hypothetical. The Classifist can do a it's basic job, but's missing all the interesting features. The Taxidermist, as I mentioned earlier, does work but it's not robust. It currently cannot handle pieces that are too long to fit in one image. However it works well enough that we will be able to actually try separating bricks and plates once the Belt Buckle goes online.

I've been working extensively on the code for the Belt Buckle, but I am learning C++ as I write it, so progress is really quite slow. It is the primary hold up right now. Currently it is capable of receiving commands and checking them for errors, storing part numbers, and reading inputs from the joystick. Not much at all, but I've erased and rewritten most of what works several times as part of the learning process and I'm very happy with how well it's turning out so far.

Once I get it to a state where it can complete a basic sorting function, we can assemble all of our prototype components into the first prototype of an actual sorting machine. In the mean time, here are a few pictures of the chaotic mess that is my work bench:

Don't worry, I'm not quitting my day job just yet.